This chapter discusses disaster recovery scenarios for the SpinServer system. A disaster is a situation in which data from one source is temporarily or permanently lost and must be restored from a backup location. An example of a disaster is a power failure on a RAID enclosure; in this case, data that is stored on the enclosure cannot be retrieved until the power is restored. If the data is duplicated on mirrors or tape backups, it can be accessed while the primary RAID enclosure is unavailable.
This section discusses the concepts that are associated with disaster recovery, including the mechanisms provided by the SpinServer system to assist with disaster recovery and considerations for designing a system that enables and simplifies disaster recovery.
The SpinServer system provides the following mechanisms for backing up data at the VFS level:
VFS SpinShots, which are point-in-time snapshots of VFSs that are automatically mounted on the source VFS. SpinShots contain only changed data and are therefore lightweight. SpinShots can be scheduled to occur at regular intervals. SpinShots are stored on the same storage pool as the source VFS; they are therefore inaccessible if the storage pool becomes unavailable. Creation of SpinShots is very fast (almost instantaneous). SpinShots are typically used to provide users with the ability to retrieve their own data in the event of an error. For instance, if a user accidentally deletes a needed file, he can retrieve it himself from a SpinShot, without needing to ask an administrator for help.
VFS mirrors, which are full copies of VFSs on a different storage pool from the source VFS. The mirror uses the same amount of storage space as the source VFS. A mirror can be scheduled to be replicate (that is, to synchronize its data with the source VFS's data) at regular intervals. Initial creation of a mirror can take some time to complete, depending on the amount of data in the source VFS, but replications are generally very fast. In the event of a disaster, a mirror can be promoted to take over for its source VFS. See Section 19.2 for information.
Tape backups of entire storage pools. SpinFS supports NDMP-compliant tape backup systems such as VERITAS and Legato. See Chapter 18 for more information.
The following table lists the differences between the various components of a complete disaster recovery setup.
Table 19-1. Comparison of Backup Types
| SpinShots | Mirrors | Tape backup | |
|---|---|---|---|
| Creation speed |
Fast SpinShots share blocks with the source VFS. |
Medium Initial creation involves copying all data in the VFS; only changed data is propagated on subsequent replications. |
Slow |
| Retrieval speed of single file |
Fast Users can retrieve their own data. |
Fast Assumes mirror is mounted at a user-visible location; otherwise, an administrator must intervene. |
N/A It is usually inefficient to restore a single file from tape. |
| Retrieval speed of entire VFS |
Medium You must copy the SpinShot, then promote the copy to read-write status. |
Fast You can remove the mirror from the group and convert it to read-write access. |
Medium Dependent on speed of tape drive and amount of data. |
| Available if primary storage pool fails? |
No SpinShots cannot be accessed if the primary VFS is offline. |
Yes Data can be restored from mirror on other storage pool. |
Yes |
Note: The performance of SpinShots and mirrors is dependent on the performance and reliability of the underlying media. For instance, Fibre Channel-backed SCSI disk arrays have better performance and greater mean time between failure (MTBF) for disk drives than do ATA disk arrays.
To ensure that you can recover data in the event of a disaster, include the following elements in the logical setup of the SpinServer system. The following scenarios assume a SpinServer system with one virtual server, two storage pools, and a tape backup system. The virtual server includes VFSs that contain user and project data.
The root VFS of the virtual server is located on one storage pool. The root VFS does not contain any data and is mirrored to the other storage pool. Because the root VFS holds no data, the mirror uses a very small amount of storage space (a few KB). The root VFS of a virtual server can always be mirrored whether or not you have the optional SpinMirror software. See Section 12.2.7 for information about mirroring the root VFS of a virtual server.
Each user VFS is scheduled to have SpinShots taken every five minutes. SpinShots are located on the same storage pool as the source VFS. If a user accidentally deletes or overwrites a file, she can retrieve it immediately from the VFS's SpinShot. For information on creating regular SpinShots, see Section 13.2.11.
Each user VFS is mirrored to another storage pool, with mirror replications scheduled to occur every 15 minutes. If the primary storage pool fails, an administrator can recover the VFSs from the secondary storage pool. Mirroring of VFSs other than the root VFS of a virtual server requires the optional SpinMirror software.
Each project VFS is also mirrored to another storage pool, with mirror replications scheduled to occur every five minutes. The read-only mirrors are made available to users who do not need to write to the project VFSs. This increases the access speed of project data to all users. Mirroring of VFSs other than the root VFS of a virtual server requires the optional SpinMirror software.
For information on mirroring VFSs, see Section 13.2.9. For information on scheduling mirror replications to occur periodically; see Section 13.2.11.
A tape backup of all data located on the virtual server is made nightly. In the event of the catastrophic loss of data on any of the disk-based storage pools, data can be restored from the tape. However, because data restoration from tape is a long process, it is recommended only as a final recourse. For information on using tape backup and restoration, see Chapter 18.
This section provides commands and procedures that can be used to recover from a catastrophic data loss. Most disaster-recovery commands are provided in the tools command node. Because the tools commands can potentially disrupt client access to data or cause data loss if used incorrectly, exercise extreme care when using them.
The tools filestorage vfs promote command promotes a mirror, a SpinShot, or a SpinShot of a mirror to the read-write parent of the VFS family. This command deletes the family's current parent read-write VFS and any SpinShots that have a data version greater than the data version of the read-only VFS that is being promoted. The command has the following syntax:
tools filestorage vfs promote -vfsname VFS_name
The following example promotes a VFS mirror named mirror.1.john to the parent read-write VFS for the VFS family:
spinserver::> tools filestorage vfs promote -vfsname mirror.1.john
The tools filestorage mirror leavegroup command removes a virtual file system (VFS) mirror from a mirror group. When a mirror is removed from the mirror group, no more manual or automatic updates from the original VFS can occur. The command has the following syntax:
tools filestorage mirror leavegroup -sourcevfsname source_VFS_name -sinkvfsname sink_VFS_name -newvfsname new_VFS_name
The following example removes a VFS mirror named vfs1_m1, which is a mirror of the VFS named vfs1, from the mirror group. The VFS is named old.vfs1.mirror after it leaves the mirror group.
spinserver::> tools filestorage mirror leavegroup -sourcevfsname vfs1 -sinkvfsname vfs1_m1 -newvfsname old.vfs1.mirror
The tools filestorage vfs deletemountpoint command deletes a VFS's mount point. Use this command if forced deletions and unmountings of VFS leave stray mount points on the parent VFS. The command has the following syntax:
tools filestorage vfs deletemountpoint -vfsname VFS_name -mountpath mount_path
The following example deletes the mount point of a VFS named bill that is mounted at /user/bill:
spinserver::> tools filestorage vfs deletemountpoint -vfsname bill -mountpath /user/bill
The tools filestorage vfs forcedelete command forces the deletion of a VFS. Use this command in the event that a VFS must be deleted when it is offline or otherwise unavailable. By default, the command attempts to unmount the VFS by deleting its mount point in its parent VFS. The VFS is then deleted whether or not the unmounting operation was successful.
The command has the following syntax:
tools filestorage vfs forcedelete -vfsname VFS_name -unmount {true | false}
The -unmount parameter specifies whether or not to attempt to unmount the VFS before deleting it. The default is true.
The following example forces the deletion of a VFS named vfs0. The command first attempts to unmount the VFS.
spinserver::> tools filestorage vfs forcedelete -vfsname vfs0 -unmount true
The tools filestorage vfs forceunmount command forces the unmounting of a VFS. Use this command in the event that the parent VFS is offline or otherwise inaccessible and a VFS needs to be unmounted. The command attempts to unmount the VFS by deleting its mount point in the parent VFS. If that attempt fails, the command clears the VFS's mount attributes, thus enabling the VFS to be mounted elsewhere. If this situation occurs, use the tools filestorage vfs deletemountpoint command to delete the VFS's mount point in the parent VFS when the parent VFS becomes available again.
The command has the following syntax:
tools filestorage vfs forceunmount -vfsname VFS_name
The following example forces the unmounting of a VFS named vfs0:
spinserver::> tools filestorage vfs forceunmount -vfsname vfs0
Commands that forcibly change a VFS's status, such as tools filestorage vfs promote, tools filestorage vfs forcedelete, and tools filestorage vfs forceunmount, can potentially leave the VFS in a state in which it is known to the storage pool but not to the management framework. Such a VFS is called a lost VFS. Data on a lost VFS is not available for client access, and normal filestorage and tools filestorage commands do not work on the VFS. To restore lost VFSs, use the tools filestorage lostnfound commands.
Note: The tools filestorage lostnfound commands must be run on the SpinServer that owns the storage pool on which the lost VFS or VFSs are located. For instance, if you want to locate all lost VFSs in a cluster, you must log in individually to each of the cluster's SpinServers and run the tools filestorage lostnfound show command.
The tools filestorage lostnfound show command displays the following information about lost VFSs:
VFS ID
Storage pool ID
VFS name
VFS type (read-write, read-only, or spinshot)
Mountpoint
Upstream VFS ID
Downstream VFS ID
VFS status (online or offline)
If the command is run with the ID and storage pool of a specific lost VFS, it displays the following additional information:
Virtual server ID
Creation time
Amount of storage used, in MB
Whether the VFS is the root VFS of its virtual server (true or false)
The command has the following syntax:
tools filestorage lostnfound show [-vfsid VFS_ID -spdevid storage_pool_ID]
The following example displays information about a lost VFS with ID 879625 on storage pool sp19:
spinserver::> tools filestorage lostnfound show -vfsid 879625 -spdevid sp19 VFS ID 879625 Storage Pool sp19 VFS lg15_vfs9.879625 Virtual Server ID 18 Type spinshot Create Time 07/20/2004 3:57 AM Mounted From Used (MB) 58 Upstream VFS ID 0 Downstream VFS ID 0 Root false Status online
Claiming a lost VFS restores it to the management framework. To claim a lost VFS, use the tools filestorage lostnfound claim command. After you run the command on a lost VFS, the data it contains is available for client access, and you can run other commands in the filestorage and tools filestorage command nodes on the VFS.
The command has the following syntax:
tools filestorage lostnfound claim -vfsid VFS_ID -spdevid storage_pool_ID
The following example restores a lost VFS with ID 879625 on storage pool sp19 to the management framework:
spinserver::> tools filestorage lostnfound claim -vfsid 879625 -spdevid sp19
You can permanently delete a lost VFS by using the tools filestorage lostnfound delete command. The command has the following syntax:
tools filestoreage lostnfound delete -vfsid VFS_ID -spdevid storage_pool_ID
The following example deletes a lost VFS with ID 788624 on storage pool sp14:
spinserver::> tools filestorage lostnfound delete -vfsid 788624 -spdevid sp14
The tools filestorage vfs makereadonly command makes the specified virtual file system read-only. This command can be used to reset the read status of a VFS that has previously been made read-write (for instance, a copy of a mirror VFS that was set to be read-write after the initial copy). The command has the following syntax:
tools filestorage vfs makereadonly -vfsname VFS_name
The following example makes a read-write VFS named vfs.wasRW a read-only VFS.
spinserver::> tools filestorage vfs makereadonly -vfsname vfs.wasRW
The tools filestorage vfs makereadwrite command can be used to make certain virtual file systems read-write. VFSs that are by definition read-only, including mirrors and SpinShots, cannot be made read-write. However, full copies of read-only VFSs (for example, a copy of a mirror) can be made read-write. The command has no effect on read-write VFSs. The command has the following syntax:
tools filestorage vfs makereadwrite -vfsname VFS_name
The following example makes a read-only VFS named vfs.wasRO a read-write VFS:
spinserver::> tools filestorage vfs makereadwrite -vfsname vfs.wasRO
The tools filestorage vfs makeroot command makes a VFS the root VFS of the virtual server. The VFS is renamed as root.virtual_server_name. For instance, if you run this command on a VFS named user that is located on a virtual server named VS1, the VFS is made the root VFS of the virtual server and is given the name root.VS1.
The command has the following syntax:
tools filestorage vfs makeroot -vfsname VFS_name
The following example makes a VFS named dept the root VFS of the virtual server:
spinserver::> tools filestorage vfs makeroot -vfsname dept
The tools filestorage vfs rename command renames a read-write VFS. The read-write VFS cannot have any associated mirrors. When a VFS is renamed, be sure to unmount it and remount it so that its parent VFS refers to it by its new name. It is recommended that you also unmount and remount any VFSs that are mounted under the renamed VFS.
The -unmount parameter specifies whether to unmount the VFS before renaming it. The default is true. In the event that the parent VFS is offline or otherwise inaccessible when a VFS is renamed, specify a value of false for the -unmount option.
The command has the following syntax:
tools filestorage vfs rename -vfsname VFS_name -newvfsname new_VFS_name -unmount {true | false}
The following example renames a VFS named old.vfs as new.vfs. The VFS is unmounted before it is renamed.
spinserver::> tools filestorage vfs rename -vfsname old.vfs -newvfsname new.vfs -unmount true
The tools filestorage vfs deletefile command deletes a specified file from a VFS. This is useful if the file is inaccessible from clients or if you do not wish the file to be accessed by clients (for example, if you suspect that the file contains a virus). The command has the following syntax:
tools filestorage vfs deletefile -vfsname VFS_name -path absolute_path_name
The following example deletes a file named /dept/eng/virus.exe from a VFS named dept.eng:
spinserver::> tools filestorage vfs deletefile -vfsname dept.eng -path /dept/eng/virus.exe
The tools filestorage vfs deletefamily command deletes a read-write VFS and all of its SpinShots, mirrors, and SpinShots of mirrors. The command has the following syntax:
tools filestorage vfs deletefamily -vfsname VFS_name
The following example deletes a VFS named vfs.parent and deletes all of its associated SpinShots and mirrors:
spinserver::> tools filestorage vfs deletefamily -vfsname vfs.parent
The tools filestorage vfs setoffline command takes a VFS offline; that is, makes it inaccessible to client access. Taking a VFS offline makes it easier to perform maintenance and disaster-recovery tasks. The command has the following syntax:
tools filestorage vfs setoffline -vfsname VFS_name
The following example takes a VFS named dept.sales offline:
spinserver::> tools filestorage vfs setoffline -vfsname dept.sales
After a VFS is taken offline by using the tools filestorage vfs setoffline command, use the tools filestorage vfs setonline command to put it online. Use this command after the necessary maintenance or disaster-recovery tasks have been performed on the VFS. The command has the following syntax:
tools filestorage vfs setonline -vfsname VFS_name
The following example puts a VFS named dept.sales online:
spinserver::> tools filestorage vfs setonline -vfsname dept.sales
If the name of a storage pool persists on the SpinServer system after the storage pool is deleted, use the tools storage pools forcedelete command to forcibly delete it. The command has the following syntax:
tools storage pools forcedelete -deviceid device_ID
The following example forcibly deletes a storage pool with the device ID sp0:
spinserver::> tools storage pools forcedelete -deviceid sp0
To force a SpinServer to take over as primary for a storage pool in a high-availability (HA) configuration, use the tools storage pools forcetakeover command. This command works only if there is currently no primary for the specified storage pool. The syntax of the command follows.
tools storage pools forcetakeover -deviceid device_ID -newprimary new_primary
The following example forces a SpinServer named Server3 to take over as primary for a storage pool named sp1.
spinserver::> tools storage pools forcetakeover -deviceid sp1 -newprimary Server3
The tools storage pools forceoffline command forcibly takes a storage pool offline. The command has the following syntax:
tools storage pools forceoffline -deviceid device_ID
The following example forces a storage pool named sp1 offline:
tools storage pools forceoffline -deviceid sp1
The tools storage units erase command to erase all data on a storage unit. This command is useful for deleting data from an evaluation system before redeploying it as a production system. The value of the -zerofill parameter specifies whether the storage unit is filled with zeroes; the default value is false. The command has the following syntax:
tools storage units erase -deviceid device_ID -zerofill {true | false}
The following example deletes all data from a storage unit named su10. The storage unit is not filled with zeroes.
tools storage units erase -deviceid su10 -zerofill false
To recover a root VFS that is a mirror of the root, perform the following steps:
Use the tools filestorage mirror leavegroup command to remove the mirror root from the mirror group. Specify a temporary name as the value of the -newvfsname parameter.
spinserver::> tools filestorage mirror leavegroup -sourcevfsname bad.root.vfs -sinkvfsname good.mirror.root -newvfsname temp.vfs
Use the tools filestorage vfs forcedelete command to delete the problematic read-write root VFS:
spinserver::> tools filestorage vfs forcedelete -vfsname bad.root.vfs -unmount true
Use the tools filestorage vfs makeroot command to make the mirror root VFS the new read-write root VFS:
spinserver::> tools filestorage vfs makeroot -vfsname temp.vfs
Unmount all VFSs that were mounted below the deleted root VFS and remount them below the new root VFS. If necessary, use the tools filestorage vfs forceunmount command to unmount the subordinate VFSs.
To recover a root VFS that is not a mirror of the root, perform the following steps:
Use the tools filestorage vfs forcedelete command to delete the problematic read-write root VFS:
spinserver::> tools filestorage vfs forcedelete -vfsname bad.root.vfs -unmount true
Use the tools filestorage vfs makeroot command to make the mirror root VFS the new read-write root VFS:
spinserver::> tools filestorage vfs makeroot -vfsname temp.vfs
Unmount all VFSs that were mounted below the deleted root VFS and remount them below the new root VFS. If necessary, use the tools filestorage vfs forceunmount command to unmount the subordinate VFSs.
To move the physical drives that comprise a storage pool from one SpinServer, perform the following steps. This procedure assumes that the physical drives comprise a single storage pool named pool0 that is being moved from a SpinServer named Server1 to a SpinServer named Server2, and that both SpinServers are in the same cluster.
If Server1 is still active, take the storage pool offline by using the storage pools setoffline command:
spinserver::> storage pools setoffline -deviceid pool0
Run the storage pools disconnect command:
spinserver::> storage pools disconnect -deviceid pool0
Physically disconnect the disk enclosure or enclosures from Server1.
Physically connect the disk enclosure or enclosures to Server2.
Run the tools storage pools forceaddpassive command:
spinserver::> tools storage pools forceaddpassive -deviceid pool0 -passivesec Server2 -priority 1
Run the tools storage pools forcetakeover command:
spinserver::> tools storage pools forcetakeover -deviceid pool0 -newprimary Server2
If desired, remove Server1 as a secondary for the storage pool by using the storage pools removesecondary command:
spinserver::> storage pools removesecondary -deviceid pool0 -secondary Server1
Virtual servers provide complete data isolation for all VFSs that it serves; a VFS on one virtual server is inaccessible from any other virtual server. If you need to move a VFS and all of its family VFSs (SpinShots and mirrors) from one virtual server to another, use the tools filestorage vfs migratefamily command. After the VFS is migrated, it cannot be accessed from its original virtual server, nor can it be used to access its original virtual server from its new virtual server.
The command has the following syntax:
tools filestorage vfs migratefamily -vfsname VFS_name -vserver virtual_server_name -force {true | false}
where
-vfsname VFS_name specifies the parent VFS that is to be migrated to a different virtual server along with its family VFSs.
-vserver virtual_server_name specifies the virtual server to which the parent VFS and its family VFSs are to be migrated.
-force {true | false} specifies whether the migration continues if an error is encountered. The default is false.
| Caution: |
|
Unless the VFS's original virtual server and its new virtual server are configured identically, with identical NIS domains, CIFS domains, and CIFS-to-NFS mappings, this command can result in extensive problems with permissions on the VFS's files, possibly causing loss of client access to data. |
To prevent problems with permissions, NetApp recommends the following procedure for migrating a VFS:
Migrate the parent VFS. The following example migrates a VFS family whose parent VFS is named dept.hr to a virtual server named vs2. The migration is not forced.
spinserver::> tools filestorage vfs migratefamily -vfsname dept.hr -vserver vs2 -force false
Mount the migrated VFS in a location that is accessible only to administrators.
Specify the use of restricted ACLs on the VFS by running the filestorage vfs modify command with the value of the -restrictedacl parameter set to true.
Set the appropriate permissions on all files in the VFS by using chmod, chown, or other operating-system tools.
If CIFS ACLs are to be used on the VFS, re-enable the use of ACLs by running the filestorage vfs modify command with the value of the -restrictedacl parameter set to false.
Unmount the VFS from its safe location, then mount it for access to users.
Copyright © 2004 Network Appliance, Inc.; copyright © 2003, 2004 Spinnaker Networks, LLC. All rights reserved. Specifications subject to change without notice.